ConformalPrediction.jl

Predictive Uncertainty Quantification in Machine Learning

Patrick Altmeyer

Delft University of Technology

July 22, 2023

Conformal Prediction

Conformal Prediction (CP) works under the premise of turning heuristic measures of Predictive Uncertainty (PU) into rigorous ones through repeated sampling or the use of calibration data.

Example: Split CP

  1. Proper training set and separate calibration set: \(\mathcal{D}_n=\mathcal{D}^{\text{train}} \cup \mathcal{D}^{\text{cali}}\).
  2. Train model on proper training set: \(\hat\mu_{i \in \mathcal{D}^{\text{train}}}(X_i,Y_i)\).
  3. Compute nonconformity scores, \(\mathcal{S}\), using calibration data \(\mathcal{D}^{\text{cali}}\) and fitted model \(\hat\mu_{i \in \mathcal{D}^{\text{train}}}\).
  4. For user-specified coverage ratio \((1-\alpha)\) compute the corresponding quantile, \(\hat{q}\), of \(\mathcal{S}\).
  5. For the given quantile and test sample \(X_{\text{test}}\), form the corresponding conformal prediction set: \(C(X_{\text{test}})=\{y:s(X_{\text{test}},y) \le \hat{q}\}\).

Blog posts

ConformalPrediction.jl

  • Fully compatible with MLJ.jl models.
  • Compatible with Flux.jl through MLJFlux.jl.
  • Many state-of-the-art CP methods implemented for regression, classification and time series modelling.

Talk Agenda

  1. 🏃 Interactive sprint of ConformalPrediction.jl (15min)
  2. 🔍 Applications (5min)
    • Conformal Chatbot
    • Conformal Image Classifier
    • Conformal Times Series Modelling
  3. 🚧 Under Construction (5min)
  4. ❓ Q&A

🔍 Applications

Conformal Chatbot

Figure 1: High-level overview of a conformalized intent classifier.

Blog post

Building a Conformal Chatbot in Julia ([blog], [TDS])

Figure 2: Demo of a REPL-based conformalized intent classifier.

Conformal Image Classifier

MNIST classifier trained using MLJFlux.jl.

Blog post

How to Conformalize a Deep Image Classifier ([blog], [TDS], [Forem])

Time Series

  • Ensemble Batch Prediction Intervals (EnbPI) proposed by Xu and Xie (2022) and recently contributed by Mojtaba Farmanbar (LinkedIn, Twitter, GitHub)
  • Residuals are used dynamically to update the nonconformity scores.

EnbPI for Victoria electricity demand dataset. Data source: MAPIE, Hyndman and Athanasopoulos (2018)

Tutorial

How to Conformalize a Time Series Model)

🚧 Under Construction

Differentiability

  • Stutz et al. (2022) introduce a smooth set size penalty to explicitly train models for efficient CP.
  • We use this in the context of gradient-based counterfactual search to obtain plausible CounterfactualExplanations.jl (currently under review).

Conformal Predictions (left), set size (centre) and smooth set size loss (right).

Contribute

Currently working on full conformal training implementation [#62].

Conformal Bayes

  1. Conformalised Bayes: simply treat Bayesian predictive posterior as our heuristic (Angelopoulos and Bates 2021).
  2. Conformal Bayes through importance sampling (Fong and Holmes 2021).

Contribute

Planning to add both ideas [#64].

🗨️ Q&A

Trustworthy AI in Julia

  1. CounterfactualExplanations.jl (JuliaCon 2022)
  2. ConformalPrediction.jl (JuliaCon 2023)
  3. LaplaceRedudx.jl (JuliaCon 2022)
  4. AlgorithmicRecourseDynamics.jl

… contributions welcome! 😊

References

Angelopoulos, Anastasios N., and Stephen Bates. 2021. “A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification.” https://arxiv.org/abs/2107.07511.
Fong, Edwin, and Chris Holmes. 2021. “Conformal Bayesian Computation.” arXiv. https://doi.org/10.48550/arXiv.2106.06137.
Hyndman, Rob J., and George Athanasopoulos. 2018. Forecasting: Principles and Practice. OTexts.
Stutz, David, Krishnamurthy Dj Dvijotham, Ali Taylan Cemgil, and Arnaud Doucet. 2022. “Learning Optimal Conformal Classifiers.” In. https://openreview.net/forum?id=t8O-4LKFVx.
Xu, Chen, and Yao Xie. 2022. “Conformal Prediction Set for Time-Series.” arXiv. https://doi.org/10.48550/arXiv.2206.07851.